Update `allow_older_versions` when running under Elastic Agent #34964

rdner · 2023-03-29T18:47:34Z

What does this PR do?

When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol.

On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later allow_older_versions update via the control protocol and the updated value was not used.

This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update.

The flag is extracted in a separate struct field for quicker access without a need to parse the configuration again.

Why is it important?

Our customers are not able to use Elastic Agent with a previous minor version of Elasticsearch, although they might be absolutely compatible.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

elastic-package stack up - this ran elasticsearch@8.6.1 for me
Build the most recent Elastic Agent (the minor version should be higher than elasticsearch)

DEV=true SNAPSHOT=true PLATFORMS="<your platform>" mage -v package

Prepare this policy file policy.yml:

outputs:
  default:
    type: elasticsearch
    log_level: debug
    enabled: true
    hosts: ["https://127.0.0.1:9200"]
    username: "elastic"
    password: "changeme"
    allow_older_versions: true
    ssl:
      verification_mode: none

inputs:
  - type: system/metrics
    id: unique-system-metrics-input
    data_stream.namespace: default
    use_output: default
    streams:
      - metricset: cpu
        data_stream.dataset: system.cpu
      - metricset: memory
        data_stream.dataset: system.memory
      - metricset: network
        data_stream.dataset: system.network
      - metricset: filesystem
        data_stream.dataset: system.filesystem

Run Elastic Agent in the standalone mode using this policy file:

sudo -- ./elastic-agent -e  run -c ./policy.yml 2> output-agent.ndjson

Before the fix you would see these errors in output-agent.ndjson despite the allow_older_versions: true in the policy:

{
  "log.level": "error",
  "@timestamp": "2023-03-29T18:41:07.937+0200",
  "message": "Failed to connect to backoff(elasticsearch(https://127.0.0.1:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.6.1, Beat=8.8.0",
  "component": {
    "binary": "filebeat",
    "dataset": "elastic_agent.filebeat",
    "id": "filestream-monitoring",
    "type": "filestream"
  },
  "log": {
    "source": "filestream-monitoring"
  },
  "log.logger": "publisher_pipeline_output",
  "log.origin": {
    "file.line": 150,
    "file.name": "pipeline/client_worker.go"
  },
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

After the fix, there are no more errors and you can see events coming in Kibana Discover.

I also removed allow_older_versions: true from the policy and ran the agent to verify that I still see the expected version validation errors when the flag is not set.
I tried to run Filebeat in a standalone mode in this configuration:

filebeat.inputs:
  - type: filestream
    id: my-filestream-id
    enabled: true
    paths:
      - "/logs/log*.log"
path.data: "/data"
logging:
  level: debug
output:
  elasticsearch:
    type: elasticsearch
    allow_older_versions: false
    hosts:
    - https://127.0.0.1:9200
    username: elastic
    password: changeme
    ssl:
      verification_mode: none

(don't forget to change the paths and write some lines to the matching log files).

And I can still see the expected error message when allow_older_versions: false:

{
  "log.level": "error",
  "@timestamp": "2023-03-29T21:21:13.492+0200",
  "log.logger": "publisher_pipeline_output",
  "log.origin": {
    "file.name": "pipeline/client_worker.go",
    "file.line": 150
  },
  "message": "Failed to connect to backoff(elasticsearch(https://127.0.0.1:9200)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.6.1, Beat=8.8.0",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

And no error message when allow_older_versions: true. Also, I see data coming in Kibana Discover on the filebeat-* index pattern.

Related issues

Closes Beats started by agent do not respect the allow_older_versions: true configuration flag #34227

When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol. On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later `allow_older_versions` update via the control protocol and the updated value was not used. This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update. The flag is extracted in a separate struct field for quicker access without a need to parse the configuration again.

elasticmachine · 2023-03-29T19:05:33Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-30T16:39:45.891+0000
Duration: 66 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	26094
Skipped	1969
Total	28063

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

rdner · 2023-03-29T19:30:07Z

The linter failure is a false-positive.

elasticmachine · 2023-03-29T19:30:56Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

ycombinator · 2023-03-30T15:43:00Z

@rdner As I'm trying to test this PR I'm seeing the following errors in output-agent.ndjson:

{"log.level":"error","@timestamp":"2023-03-30T08:37:07.850-0700","message":"Exiting: failed to get host information: 1 error: could not get FQDN, all methods failed: failed looking up CNAME: lookup cowchicken: no such host: failed looking up IP: lookup cowchicken: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-30T08:37:07.850-0700","message":"Exiting: failed to get host information: 1 erro
r: could not get FQDN, all methods failed: failed looking up CNAME: lookup cowchicken: no such host: failed looking up IP: lookup cowchicken: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-03-30T08:37:07.850-0700","message":"Exiting: failed to get host information: 1 error: could not get FQDN, all methods failed: failed looking up CNAME: lookup cowchicken: no such host: failed looking up IP: lookup cowchicken: no such host","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}

As such, none of the component Beats are starting up.

I am very familiar with these errors 😅; they were fixed very recently via #34946. If you rebase this PR on main, that should make these errors go away.

I'm working around this error locally but just mentioning the fix here for anyone else who might run into it while testing this PR.

rdner · 2023-03-30T15:50:13Z

@ycombinator hmm, I've tested with Beats on this branch, your agent must be newer, so the behaviour is different.

I don't feel like merging main and waiting for the CI all over again :)
Can you workaround the error and still test it?

ycombinator · 2023-03-30T15:52:53Z

hmm, I've tested with Beats on this branch, your agent must be newer, so the behaviour is different.

The error will only happen if your hostname is set to something that cannot be resolved via DNS. Mine is deliberately set that way 🙂. So it's likely no one else is going to run into this issue. Just wanted to mention it in case someone else does.

I don't feel like merging main and waiting for the CI all over again :) Can you workaround the error and still test it?

Yup, understandable! I'm working around it locally and testing.

libbeat/cmd/instance/beat.go

ycombinator · 2023-03-30T16:34:52Z

Code LGTM (and thanks for the tests). I'm having some issues testing this locally but that might just be due to my local setup. Working through it...

ycombinator

LGTM.

Tested with instructions in PR and I can get the Beats under Agent (8.8.0-SNAPSHOT) to connect to an older ES (8.6.1):

{"log.level":"info","@timestamp":"2023-03-30T10:25:58.007-0700","message":"Attempting to connect to Elasticsearch version 8.6.1","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.logger":"esclientleg","log.origin":{"file.line":291,"file.name":"eslegclient/connection.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-30T10:25:58.080-0700","message":"Attempting to connect to Elasticsearch version 8.6.1","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.origin":{"file.line":291,"file.name":"eslegclient/connection.go"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"esclientleg","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-30T10:25:58.196-0700","message":"Connection to backoff(elasticsearch(https://test-older-versions.es.us-central1.gcp.qa.cld.elstc.co:9243)) established","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":147,"file.name":"pipeline/client_worker.go"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-03-30T10:25:58.267-0700","message":"Connection to backoff(elasticsearch(https://test-older-versions.es.us-central1.gcp.qa.cld.elstc.co:9243)) established","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":147,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Also tested with allow_older_versions line removed from config and the connection to ES fails, as expected:

{"log.level":"error","@timestamp":"2023-03-30T10:30:19.289-0700","message":"Failed to connect to backoff(elasticsearch(https://test-older-versions.es.us-central1.gcp.qa.cld.elstc.co:9243)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.6.1, Beat=8.8.0","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol. On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later `allow_older_versions` update via the control protocol and the updated value was not used. This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update. (cherry picked from commit 1a9d627)

…er Elastic Agent (#34979) * Update `allow_older_versions` when running under Elastic Agent (#34964) When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol. On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later `allow_older_versions` update via the control protocol and the updated value was not used. This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update. (cherry picked from commit 1a9d627) --------- Co-authored-by: Denis <denis.rechkunov@elastic.co>

When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol. On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later `allow_older_versions` update via the control protocol and the updated value was not used. This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update. (cherry picked from commit 1a9d627) # Conflicts: # libbeat/cmd/instance/beat_test.go

…er Elastic Agent (#35574) * Update `allow_older_versions` when running under Elastic Agent (#34964) When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol. On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later `allow_older_versions` update via the control protocol and the updated value was not used. This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update. (cherry picked from commit 1a9d627) # Conflicts: # libbeat/cmd/instance/beat_test.go * Remove duplicate changelog entry * Fix conflict in beat_test.go. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Craig MacKenzie <craig.mackenzie@elastic.co>

When Beats are running under Elastic Agent their initial output configuration is empty. Only a few moments later the output configuration arrives as an update via the control protocol. On startup Beats register a global Elasticsearch connection callback to validate the Elasticsearch version. Unfortunately, this callback didn't account for the later `allow_older_versions` update via the control protocol and the updated value was not used. This fixes that broken behaviour and makes an update to the entire in-memory output configuration on each control protocol update.

rdner added bug Filebeat Filebeat backport-v8.7.0 Automated backport with mergify labels Mar 29, 2023

rdner self-assigned this Mar 29, 2023

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2023

rdner added 2 commits March 29, 2023 20:57

Add changelog entry

c548f24

Move changelog entry to the right place

04159b3

rdner marked this pull request as ready for review March 29, 2023 19:30

rdner requested a review from a team as a code owner March 29, 2023 19:30

rdner requested review from cmacknz and leehinman and removed request for a team March 29, 2023 19:30

rdner added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Mar 29, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2023

Fix panic in tests

9eb4c54

jlind23 requested review from faec and belimawr March 30, 2023 07:02

ycombinator reviewed Mar 30, 2023

View reviewed changes

libbeat/cmd/instance/beat.go Outdated Show resolved Hide resolved

rdner added 2 commits March 30, 2023 18:24

Merge branch 'main' into fix-allow-older-versions

f62cbd2

Remove the cached flag

69f1fa7

rdner added 2 commits March 30, 2023 18:37

Remove unnecessary test, add more checks

304aae3

Improve output type handling

3898778

ycombinator approved these changes Mar 30, 2023

View reviewed changes

rdner merged commit 1a9d627 into elastic:main Mar 30, 2023

rdner deleted the fix-allow-older-versions branch March 30, 2023 18:44

mergify bot mentioned this pull request Mar 30, 2023

[8.7](backport #34964) Update allow_older_versions when running under Elastic Agent #34979

Merged

cmacknz added the backport-v8.6.0 Automated backport with mergify label May 25, 2023

mergify bot mentioned this pull request May 25, 2023

[8.6](backport #34964) Update allow_older_versions when running under Elastic Agent #35574

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `allow_older_versions` when running under Elastic Agent #34964

Update `allow_older_versions` when running under Elastic Agent #34964

rdner commented Mar 29, 2023 •

edited

Loading

elasticmachine commented Mar 29, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

rdner commented Mar 29, 2023

elasticmachine commented Mar 29, 2023

ycombinator commented Mar 30, 2023 •

edited

Loading

rdner commented Mar 30, 2023

ycombinator commented Mar 30, 2023

ycombinator commented Mar 30, 2023

ycombinator left a comment •

edited

Loading

Update allow_older_versions when running under Elastic Agent #34964

Update allow_older_versions when running under Elastic Agent #34964

Conversation

rdner commented Mar 29, 2023 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Mar 29, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

rdner commented Mar 29, 2023

elasticmachine commented Mar 29, 2023

ycombinator commented Mar 30, 2023 • edited Loading

rdner commented Mar 30, 2023

ycombinator commented Mar 30, 2023

ycombinator commented Mar 30, 2023

ycombinator left a comment • edited Loading

Choose a reason for hiding this comment

Update `allow_older_versions` when running under Elastic Agent #34964

Update `allow_older_versions` when running under Elastic Agent #34964

rdner commented Mar 29, 2023 •

edited

Loading

elasticmachine commented Mar 29, 2023 •

edited by jenkins-beats-ci bot

Loading

ycombinator commented Mar 30, 2023 •

edited

Loading

ycombinator left a comment •

edited

Loading